Traditionally, different methods have been used for information collection and management of networks and devices. Most popular is the older Simple Network Management Protocol (SNMP). However, while SNMP is extensible and can incorporate new management elements, it does not give finer flow information such as at the HTTP transaction level. Instead, SNMP provides only aggregated device-level information.
Similarly, the newer IP Flow Information Export, or IPFIX, protocol also focuses on lower level flow information. IPFIX is a universal standard for exporting Internet Protocol flow information from routers, switches, probes, and other intermediary devices. IPFIX is defined in Internet Engineering Task Force (IETF) RFCs 5101 and 5102, the latter of which includes over 200 standard information elements. According to RFC 5101, a flow is defined as a set of IP packets passing an observation point in the network during a certain time interval, with packets belonging to a particular flow having common properties. As noted in RFC 5101, however, the flow definition does not necessarily match application-level end-to-end streams.
IPFIX includes features such as template based flow information definition and extensibility options. Key fields are also dynamically defined along with template definitions for particular information. Each template defines an individual flow data record and its key fields, and may contain a set of standard information elements (IE) and enterprise specific information elements (EIE) and their order in the corresponding data record. A data record may be linked to a template using a template ID.
IETF RFC 5470, “Architecture for IP flow information export,” defines three key components of the IPFIX architecture. The first is an exporter process, which collects, filters, and/or samples the required flow information and exports. The second component is the collector process, which reads the templates and key fields and distinguishes different flow records and collects them for later consumption from the users. The third component is the users, which interact with the collector process to get the required flow records and process the information to determine intelligent decision points.
IPFIX was originally developed by Cisco Systems, Inc., as the proprietary Netflow protocol. Netflow and IPFIX describe extensible records for describing layer 2 and layer 3 network flows. Typically, these include values aggregated from multiple higher layer communications, and accordingly, fail to address session or application layer transaction boundaries or other fine detail.
Other techniques exist for capturing application layer network information, but each have drawbacks. For example, HTML injection and logging lacks flexibility and extensibility in adding new elements, and requires proprietary applications on both ends of a communication in order to capture data. Port mirroring provides fine detail, but is highly demanding of both network bandwidth and CPU usage.